Electronic image sensors, such as charge-coupled devices (CCDs), complementary metal oxide semiconductor (CMOS) sensors, and photodiode arrays, are designed to capture image information by converting a spatially varying electromagnetic exposure to a spatially varying analog and/or digital signal. In an ideal sensor, the final signal is due only to the distribution of electromagnetic (EM) energy collected at the image plane. Real sensors, however, are also susceptible to thermal excitation. The result of the thermal excitation is additive noise—false signal components due to non-image energy within the sensor. All sensors, from the human retina to electronic sensor arrays, suffer from this effect. Signals from electronic sensors are typically amplified, exacerbating the effect.
Referring to FIG. 1, a thirty second exposure captured on a CCD sensor in a Kodak Professional DCS 315 digital-camera is illustrated. The additive noise in this digital image is clearly visible. It is also evident that the noise is not equally distributed in spectral content.
While thermal excitation in the sensor array and subsequent processing are responsible for the additive noise, the final effect is due to several interacting influences. Because of image sensors' varying sensitivity to different wavelength bands, system designs often include differential gain for each channel, leading to more severe noise characteristics in the channels with higher gain. Many color sensors have color filter arrays overlaying the sensor surface. These arrays often have different spatial sampling frequencies in the different color planes; the sampling density is typically highest in the green to optimize luminance-based resolution. One example of this is disclosed in U.S. Pat. No. 3,971,065, which is hereby incorporated by reference.
The additive noise varies with several variables, notably sensor temperature, exposure duration, and gain. As the sensor temperature is raised, thermal excitation within the sensor increases, leading to more pronounced noise levels. The duration of the exposure also plays an important role in the degree to which noise degrades image quality.
Referring to FIGS. 2A-2F, the central region from a sequence of exposures ranging from 1 through 30 seconds is shown (the camera aperture was changed to maintain equal exposures throughout the sequence). The effect of thermal noise on images captured with electronic sensors places practical limits on exposure duration without cooling. Current digital cameras restrict the exposure duration and/or caution users that long exposures may lead to a degradation of image quality. By way of example, the users guide for the Kodak DCS-315 digital camera warns that exposure durations greater than 0.25 seconds will lead to reduced image quality.
Noise in the final image is also a function of detector sensitivity and gain. Increasing the effective sensitivity of a system by, for example, increasing system gain, generally leads to increased noise. The resultant noise levels limit the practical sensitivity of modern digital cameras. Increasing the sensitivity has commercial value because the cameras can be used in lower illumination levels and/or with shorter exposure times without electronic flash and the maximum distance at which a given flash is effective is increased. A technique for reducing apparent noise in digital imaging systems would enhance the value of digital camera systems and speed their acceptance for applications ranging from consumer photography to forensic imaging (for example).
Another variable affecting the noise characteristics in a digital imaging system is the sensor array itself. Specific pixels are subject to varying degrees of noise due to thermal and amplification variations. This characteristic is unlike detecting ‘bad pixels’ (or ‘dark pixels’ or other term) which vary in sensitivity. One example of an apparatus for correcting faulty pixel signals by replacing the faulty pixel signals with normal pixel signals is disclosed in U.S. Pat. No. 5,392,070, which is hereby incorporated by reference. Methods have also been described to correct for pixels' fixed offset such as in U.S. Pat. No. 4,739,495, which is hereby incorporated by reference.
Pixels referred to in this invention are working sensor elements, and contribute fully to exposures made at relatively short exposures and/or processed at low amplification levels. They differ from the plurality of pixels by their susceptibility to noise.
The traditional approach to reducing the effects of additive noise in digital images is the image processing technique known as median filtering. One example of median filtering is disclosed in U.S. Pat. No. 4,783,753, which is hereby incorporated by reference. Such filtering techniques have also been applied to noise due to dust and scratches in images as disclosed, by way of example, in U.S. Pat. No. 5,036,405, which is hereby incorporated by reference.
Other systems have been devised to reduce noise in systems by affecting different pixels differently, such as in U.S. Pat. No. 4,858,013, which is hereby incorporated by reference, but none of these systems make use of information regarding pixels' fixed susceptibility to noise. The success of the system described here is due to the filtering method, as well as the identification and mapping of image elements based on their noise susceptibility under varying exposure/processing conditions. The median filter operation replaces each pixel with the median value of the pixels that surround a particular pixel within a specified radius. The traditional median filtering technique also applies the same filter to every pixel in the red, green, and blue channels.
Unfortunately, the three-color sensors have different noise characteristics in each channel. This occurs for several reasons, chiefly because of the non-uniform spectral sensitivity and the effect of the subtractive filter array. Because the three channels have different characteristics, the optimum median filter for each channel is not likely to be the same for each channel.
Referring to FIGS. 3A-3D, the effect of median filtering the fifteen second exposure shown in FIG. 2E is shown. While it is evident that median filtering with progressively larger radii is effective in reducing additive noise, this approach also degrades image quality to a degree that makes the image unacceptable.